Crowd based content delivery

ABSTRACT

Methods and systems for crowd based content delivery are provided. According to one embodiment, a preference of a content publisher is received at a resource manager. An indication is received at the resource manager that the resource manager is to redirect a request for a content item. A state of a network is monitored by the resource manager. A resource provider in the network capable of servicing the request is selected based at least in part on the state of the network and the preference of the content publisher. The request is redirected to the resource provider.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/655,900, filed Jan. 7, 2010, which claims the benefit of priority toU.S. Provisional Application No. 61/269,646, filed on Jun. 25, 2009,both of which are hereby incorporated by reference in their entirety forall purposes.

COPYRIGHT NOTICE

Contained herein is material that is subject to copyright protection.The copyright owner has no objection to the facsimile reproduction ofthe patent disclosure by any person as it appears in the Patent andTrademark Office patent files or records, but otherwise reserves allrights to the copyright whatsoever. Copyright © 2009-2013, Fortinet,Inc.

BACKGROUND

1. Field

Embodiments of the present invention generally relate to contentdelivery. In particular, embodiments of the present invention relate tocrowd based content delivery infrastructure and methods.

2. Description of the Related Art

Many network entities have excess computing resources that are unusedand wasted. It would be useful to create a marketplace for theseotherwise untapped resources.

SUMMARY

Methods and systems are described for crowd based content delivery.According to one embodiment, a request handling server obtains a ruleset for managing the traffic of a content publisher. A requestassociated with the content publisher is received at the requesthandling server. When the received request is a content request,directly servicing the received request or redirecting the receivedrequest by the request handling server to another server capable ofhandling the request based on the rule set. When the received requestcomprises a Domain Name System (DNS) request, responding to the DNSrequest, by the request handling server, with a DNS response based onthe rule set.

Other features of embodiments of the present invention will be apparentfrom the accompanying drawings and from the detailed description thatfollows.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings and in which like reference numerals refer to similar elementsand in which:

FIG. 1 is a block diagram illustrating an embodiment of a networkenvironment for operating a crowd based computing platform in accordancewith an embodiment of the present invention.

FIGS. 2A-2B illustrate processes for adding a resource provider to anetwork managed by a resource manager in accordance with embodiments ofthe present invention.

FIGS. 3A-3B illustrate processes for employing a resource providerconfigured as a proxy server to service a content request in accordancewith embodiments of the present invention.

FIGS. 4A-4B illustrate processes for adding a resource consumer to anetwork managed by a resource manager in accordance with embodiments ofthe present invention.

FIG. 5A is a block diagram illustrating a resource manager in accordancewith an embodiment of the present invention.

FIG. 5B illustrates a process for redirecting a request for a contentitem to a resource provider in accordance with an embodiment of thepresent invention.

FIGS. 6A-6C are block diagrams illustrating manners in which a contentrequest from an end user are redirected by the resource manager to anode capable of servicing the request in accordance with embodiments ofthe present invention.

DETAILED DESCRIPTION

Methods and systems are described for crowd based content delivery.Embodiments of the present invention can be implemented in numerousways, including as a process; an apparatus; a system; a composition ofmatter; a computer program product embodied on a computer readablestorage medium; and/or a processor, such as a processor configured toexecute instructions stored on and/or provided by a memory coupled tothe processor. In this specification, these implementations, or anyother form that the invention may take, may be referred to astechniques. In general, the order of the steps of disclosed processesmay be altered within the scope of the invention. Unless statedotherwise, a component such as a processor or a memory described asbeing configured to perform a task may be implemented as a generalcomponent that is temporarily configured to perform the task at a giventime or a specific component that is manufactured to perform the task.As used herein, the term ‘processor’ refers to one or more devices,circuits, and/or processing cores configured to process data, such ascomputer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims,and the invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example, andthe invention may be practiced according to the claims without some orall of these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIG. 1 is a block diagram illustrating a network environment 100 foroperating a crowd based computing platform in accordance with anembodiment of the present invention. Resource manager or aggregator 102facilitates transactions between resource consumers 104 and resourceproviders 106. The various entities comprising network environment 100communicate via network 108, which may comprise any public or privatenetwork such as a LAN, WAN, the Internet, etc., using any appropriatecommunication protocol such as HTTP (Hypertext Transfer Protocol), SSL(Secure Sockets Layer), RTMP (Real Time Messaging Protocol), RTMP-E(Encrypted Real Time

Messaging Protocol), RTMP over HTTP, torrent style protocols, etc.Resource manager 102 manages anyone or more types of computing resourcesfor performing tasks such as, but not limited to, content distributionor delivery, parallel processing, security, storage, etc. Althoughdepicted as a single entity in network environment 100, resource manager102 may comprise a plurality of interconnected computing systems thatperform the various tasks associated with managing resources. Resourceproviders 106 comprise a crowd of members who donate or sell theirresources to resource consumers 104. Any type of member may be a part ofthe crowd of resource providers 106 such as individuals, groups,corporations, universities, content delivery networks (CDNs), internetservice providers (ISPs), carriers, cloud computing networks, serverfarms, etc. Resource manager 102 handles various processes associatedwith the exchange of resources between resource providers 106 andresource consumers 104 such as monitoring performance, trackingstatistics, enforcing provider and consumer preferences, providingsecurity, billing, etc. Although a single resource manager 102 isdepicted in network environment 100, a plurality of networked resourcemanagers may be deployed across various geographical regions, e.g., tomanage resource providers and resource consumers across the world.

As further described herein, the resource manager establishes andmanages a confederation of resource providers. A resource provider maycomprise any network node that has extra capacity that can be providedor sold to consumers who heed the resource. A resource provider, forexample, may set a price for an available resource, and if a consumerfinds that resource attractive, the consumer pays for the right to useit. In some embodiments, the resource manager assists a resourceprovider in becoming a CDN or CDN node that is capable of deliveringcontent on behalf of a content publisher, i.e., the resource consumer inthis case. In some such cases, for example, the resource managerprovides configuration software which when installed on one or moreservers of the resource provider configures the servers to behave as CDNnodes. A resource provider configured as a CDN node may be employed toserve content based on the availability of resources at the node, whichmay vary based on factors such as current load, day and time, geographiclocation, etc.

Although many of the given examples are with respect to crowd basedcontent delivery, the techniques described herein may be employed withrespect to any computing resource and/or task. The described techniquesallow excess capacity of resource providers that is otherwise unused andwasted to ‘be utilized and/or monetized. For example, even though manyISP and carrier networks are bidirectional (e.g., a 10 Gigabitconnection comprises 10 Gigabit inbound and 10 Gigabit outbound), theytypically have significantly more inbound traffic due to usersdownloading content than outbound traffic due to relatively fewer usersuploading content, resulting in a large amount of unused outboundbandwidth. These networks typically only pay for one direction oftraffic, either ingress or egress, whichever is greater. Since inboundtraffic usually eclipses outbound traffic, the networks commonly have alarge amount of free idle outbound bandwidth. It would be useful, forexample, to add servers at these networks and configure them as CDNnodes so that the extra available outbound bandwidth can be utilizedand/or monetized using the techniques described herein.

FIGS. 2A-2B illustrate processes for adding a resource provider to anetwork managed by a resource manager in accordance with embodiments ofthe present invention. In some cases, the processes may be employed toconfigure a network node to become a CDN node capable of deliveringcontent on behalf of a content publisher. Process 200 is employed by aresource provider such as any of resource providers 106 of FIG. 1 orresource provider 608(a) of FIG. 6A; and process 202 is employed by aresource manager such as resource manager 102 of FIG. 1, resourcemanager 500 of FIG. 6A, or resource manager 602 of FIGS. 6A-6C. Process200 starts at 204 at which a resource provider sets up an account withthe resource manager, e.g., via a web site and/or interface madeavailable by the resource manager. Various parameters that define theterms and conditions with respect to which the resource provider iswilling to provide resources or services are specified with respect tothe resource provider's account. For example, the types of resourcesand/or the services that the resource provider is willing to deliverand/or perform may be specified. In the cases in which it is desired toconfigure the resource provider as a CDN node, for instance, a proxyserver option is selected. The specification of resources may furtherinclude a specification of the amount or percentage of a total resource(e.g., hard drive, memory, CPU, network bandwidth, etc.) to be madeavailable, which may be specified as a function of time. Moreover,resource consumers and/or types of resource consumers may be specified,e.g., on an inclusion or exclusion basis. The various entities permittedto use and/or excluded from using the resource provider's resources maybe specified or selected by name and/or by the nature of their business.For example, in some cases, any nonprofit organization may be permittedto use the resource provider's resources while in other cases only oneor more specified entities may be permitted. Similarly, content typesfor using the resources or services with may be specified, e.g., on aninclusion or exclusion basis. For example, in some cases, any contenttype other than adult content may be specified as permissible.

Furthermore, prices at which the resource provider is willing to provideresources or services may be specified. Different prices may bespecified based on different criteria such as consumer or consumer type,content type, time of use, etc. For instance, resources may be donatedor provided for free to nonprofit organizations but charged a specifiedprice per unit from other consumers and different prices may bespecified for different consumers; higher prices may be specified foruse of resources with respect to certain types of content such as adultcontent; prices may be specified as functions of time and/or date, e.g.,higher prices may be specified during business hours when the resourceprovider has peak loads than during nights, weekends, and holidays; etc.Other information such as the geographic location of the resources,environmental considerations such as the carbon footprint for providinga resource or service, etc., may also be specified. The variousparameters described may be separately specified for each machine orserver available at the resource provider. The account informationprovided at 204 is received by the resource manager at 206 of process202. In some embodiments, steps 204 and 206 include the resourceprovider acquiring and being granted by the resource manager a resourceprovider identifier and key or password via which the resourceprovider's account with the resource manager may be accessed. In variousembodiments, the parameters and information described above as beingspecified with respect to a resource provider's account may be specifiedduring initial registration or at a later time and may be later updatedor changed as applicable. Other parameters in addition to and/or insteadof those described may be specified as applicable.

Software for configuring a node as a resource provider made available bythe resource manager at 208 is downloaded and installed by the resourceprovider at 210. In various embodiments, the software may comprise anapplication, an operating system, a server instance, such as a JavaVirtual Machine, a specialized C proxy application, such as Varnish thatruns on a server, a plug-in for a browser or any other application orinterface, etc. One or more parameters or preferences specified withrespect to the resource provider's account with the resource manager maybe retrieved during installation of the software at 210. In someembodiments, one or more parameters or preferences may be specifiedduring installation at 210 rather than during step 204 as describedabove. The configuration software is installed at 210 on each computeror machine desired to be configured as a resource provider. At 212, thesoftware installed on a machine appropriately configures, the machine asa resource provider based on the preferences specified. For example, amachine may be configured by the software at 212 to function as acaching proxy server. In some embodiments, the software conducts one ormore performance tests at 212 to assess the quality of the availableresources. For example, performance tests on the hard drive, memory,CPU, network connections, etc., may be performed. Once a node has beenconfigured as a resource provider at 212, an indication that theconfiguration is complete is communicated by the software and receivedby the resource manager at 214. In some embodiments, the resourcemanager receives at 214 the results of the performance tests conductedat 212. The results of the performance tests are reported to theresource manager so that the resource manager can appropriately marketand provide or sell the resources to resource consumers. In someembodiments, performance tests are periodically conducted by thesoftware at the resource provider and reported back to the resourcemanager so that the resource manager is always aware of changes inperformance levels and can make appropriate use of the resources of theresource provider. At 216, the resource provider is added to the networkmanaged by the resource manager.

FIGS. 3A-3B illustrate processes for employing a resource providerconfigured as a proxy server to service a content request in accordancewith embodiments of the present invention. Process 300 is employed by aresource manager such as resource manager 102 of FIG. 1, resourcemanager 500 of FIG. 5A, or resource manager 602 of FIGS. 6A-6C; andprocess 302 is employed by a resource provider, such as any of resourceproviders 106 of FIG. 1 or resource provider 608(a) of FIG. 6A. Process300 starts at 304 at which a content request is received by the resourcemanager from a user. The requested content, for example, is published bya content publisher that uses content delivery resources and is theresource consumer in this case. In this example, the content publishersubscribes to the services of the resource manager for facilitating theservicing of requests for content published by the content publisher,and the requests for the publisher's content from individual users aredirected or redirected to the resource manager. At 306, the contentrequest received at 304 is redirected by the resource manager, e.g., viaan HTTP 302 redirect (or similar content redirect action in otherprotocols such as RTMP), to a resource provider capable of servicing therequest and permitted by the resource consumer to service the request.The redirected content request is received by the resource provider at308, and the requested content is served by the resource provider to theuser who requested the content at 310.

In some embodiments, the requested content is already cached at theresource provider when it receives the request at 308. In otherembodiments, the resource provider obtains and caches the requestedcontent from an origin of the content publisher in response to receivingthe request at 308. Log data of the transaction, which includesinformation associated with delivering the requested content (such asthe file identifier, timestamp of delivery, the source and/ordestination IP addresses, the file size, the bandwidth consumed, theprice or cost for delivery, etc.), is compiled by the software at theresource provider and communicated to the resource manager at 312. Insome embodiments, the log data at least in part comprises a W3C webserver log. The log data is received by the resource manager at 314. Insome embodiments, the log data received at 314 is parsed, aggregated,and/or stored at the resource manager and may, for example, be lateremployed for billing the associated resource consumer and reimbursingthe resource provider.

FIGS. 4A-4B illustrate processes for adding a resource consumer to anetwork managed by a resource manager in accordance with embodiments ofthe present invention. Process 400 is employed by a resource consumer,such as any of resource consumers 104 of FIG. 1; and process 402 isemployed by a resource manager, such as resource manager 102 of FIG. 1,resource manager 500 of FIG. 5A, or resource manager 602 of FIGS. 6A-6C.Process 400 starts at 404 at which a resource consumer sets up anaccount with the resource manager, e.g., via a web site and/or interfacemade available by the resource manager. Various parameters that definethe terms and conditions with respect to which the resource consumer iswilling to use or purchase resources or services are specified withrespect to the resource consumer's account. For example, the types ofresources and/or services that the resource consumer desires to use orpurchase as well as the types of content with respect to which theresource consumer expects to use the resources and services may bespecified. For instance, the resource consumer may sign up for contentdelivery services for serving video content. Furthermore, other criteriasuch as the required quality or performance levels, the requiredsecurity levels, etc., may be specified. In addition, resource providersand/or resource provider types may be specified, e.g., on an inclusionor exclusion basis. The various entities permitted to provide and/orexcluded from providing resources or services to the resource consumermay be specified or selected by name and/or by type. For example, acontent publisher seeking content delivery services may not allowuntrusted members of the crowd such as random individual users to servetheir content but may allow trusted entities, such as prominent ISPs orcarriers, to serve their content. Moreover, prices or price ranges thatthe resource consumer is willing to pay for various resources andservices may be specified. Different prices may be specified based ondifferent criteria such as resource provider or provider type, contenttype, time of use, etc. Other information such as geographic location,environmental considerations such as the permissible carbon footprintfor obtaining a resource or service, etc., may also be specified. Thevarious parameters described may be separately specified for eachresource or service or type of resource or service desired by theresource consumer.

The account information provided at 404 is received by the resourcemanager at 406 of process 402. In some embodiments, steps 404 and 406include the resource consumer acquiring and being granted by theresource manager a resource consumer identifier and key or password viawhich the resource consumer's account with the resource manager may beaccessed. In various embodiments, the parameters and informationdescribed above as being specified with respect to a resource consumer'saccount may be specified by the resource consumer during initialregistration or at a later time and may be later updated or changed asapplicable. Other parameters in addition to and/or instead of thosedescribed may be specified by the resource consumer as applicable. Withrespect to a content publisher signing up for content delivery services,for example, content origin locations where the content is publishedmaybe specified; and/or CDN providers with which the content publisherhas contracts, if any, and the terms of those contracts may be specifiedso that those CDN providers may be used to service content requests. Theresource consumer is added to the network managed by the resourcemanager at 408. Once the resource consumer subscribes with the resourcemanager for a particular resource and/or service, needs for thatresource and/or service are directed by the resource consumer to theresource manager at 410. With respect to content delivery, for example,when a content publisher subscribes to the services of the resourcemanager for servicing content requests, the content publisher ensuresthat user requests for content published by the content publisher aredirected or redirected to the resource manager. Resource consumer needsare, in turn, directed for servicing to appropriate resource providersin the network by the resource manager at 412 based on the preferencesspecified by the resource consumer. In some embodiments, a network nodemay sign up both as a resource provider and a resource consumer for thesame or different resources, e.g., using process 200 and 202 of FIGS.2A-2B and processes 400 and 402 of FIGS. 4A-4B.

The resource manager comprises one or more networked modules, each ofwhich may comprise one or more hardware and/or software components. FIG.5A is a block diagram illustrating a resource manager 500 in accordancewith an embodiment of the present invention. For example, resourcemanager 500 may comprise resource manager 102 of FIG. 1 or resourcemanager 602 of FIGS. 6A-6C. In the depicted embodiment, resource manager500 comprises a monitoring module 502 and a director module 504.Monitoring module 502 monitors the health of the network and variousnodes. Data may be received by or input into monitoring module 502 froma variety of sources, e.g., on the Internet. Different types of data maybe input by different sources depending on the types of data availableto them. In some embodiments, the data comprises performance statisticsassociated with quality of service and end user experience such asaverage and/or maximum throughput, DNS lookup time, time to firstconnection, download time, etc. In some cases, dedicated monitoringservers may be placed across the network that report back variousperformance characteristics. In some cases, log data compiled byresource providers at the conclusion of each transaction and/or logs ofopen proxy servers may be input into monitoring module 502. In somecases, plug-ins may be included in sources, such as web browsers, mediaplayers, download agents, etc., that ping back the performancestatistics available to them. The data received by monitoring module 502is parsed, analyzed, aggregated, and/or stored. In some embodiments,various entities may desire to obtain and/or purchase performancestatistics on their nodes and/or networks. In such cases, the relevantdata compiled by monitoring module 502 may be presented or reported,e.g., via a dashboard, to the entity” i.e., the resource consumer inthis case. For example, a CDN may be interested in monitoring the healthof its nodes so that a prescribed quality of service can be maintained,and a server farm may find a real time dashboard providing statistics oninbound and outbound traffic useful.

Director module 504 receives requests for resources or services andselects appropriate resource providers to service the requests based onthe preferences specified by the resource consumers and resourceproviders. In some embodiments, decisions for selecting resourceproviders are made by director module 504 based at least in part on thedata collected and/or information learned by monitoring module 502. Withrespect to content delivery, for example, if a portion of a CDN in aparticular geographical region goes down, existence of the black spot(i.e., a poorly performing area in a network or geography) in the CDN isquickly learned by monitoring module 502 and communicated to directormodule 504 so that content requests are not redirected by directormodule 504 to at least those nodes of the CDN. A prescribed quality ofservice and user experience is maintained in the network managed bytraffic manager 500 by making decisions based on the current state ofthe network and its constituent nodes as determined by monitoring module502. In some embodiments, monitoring module 502 includes a spiderprocess that monitors the requests coming into resource manager 500 andthat crawls the network managed by resource manager 500 to determineand/or report the availability of various resource providers to serviceincoming requests. With respect to content delivery, for example, thespider learns and stores the locations of content items (i.e., files) inthe network. For instance, the spider may interrogate a CDN using anappropriate interrogation methodology (e.g., an HTTP HEAD request orsimilar request in RTMP or other protocols) to determine theavailability of a particular content item at the CDN. The spider mayalso coordinate pre-fetching of a content item at a node to warm thecache at the node before a request for that content item is redirectedto the node by director module 504. Thus, the spider assists directormodule 504 in directing a request to a resource provider capable ofservicing the request. In some embodiments, decisions for selectingresource providers are made by director module 504 based at least inpart on past traffic redirected to the resource providers, e.g., toprevent any given resource provider from becoming overloaded and/or toload balance a plurality of available resource providers. In someembodiments, information associated with the requests redirected bydirector module 504 (such as the resource requested, the user and/orresource consumer issuing the request, the resource provider selected toservice the request, the type and amount of resources expected toservice the request, the resource provider price for servicing therequest, etc.) may be logged and stored at the resource manager andlater employed, e.g., to generate statistics or for billing purposes.

FIG. 5B illustrates a process for redirecting a request for a contentitem to a resource provider in accordance with an embodiment of thepresent invention. Process 506 is employed by a resource manager, suchas resource manager 102 of FIG. 1, resource manager 500 of FIG. 5A, orresource manager 602 of FIGS. 6A-6C. In various embodiments, process 506may be employed in anticipation of receiving a request for a contentitem or in response to receiving a request for a content item. Process506 starts at 508 at which an indication that the resource manager is toredirect a request for a particular content item to a resource providercapable of servicing the request is received. In some embodiments, theindication is received from the publisher of the content item, i.e., theresource consumer in this case, such as via the publisher's account withthe resource manager or via learning from the publisher origin thecontent published by the publisher. In some embodiments, the indicationis received in response to a (previous) request for that content itembeing forwarded or redirected to and/or received by the resourcemanager. At 510, an appropriate resource provider capable of servicingrequests for the content item is selected, e.g., based on thepreferences specified by the content publisher. In some embodiments, theselected resource provider comprises a network node configured as aproxy server, e.g., via process 200 of FIG. 2A. In some embodiments, theselected resource provider comprises a CDN. In some such cases, thecontent publisher may have specified using the CDN to service requests,e.g., based on the terms of an existing contract with the CDN. In someembodiments, a resource provider able to service requests for thecontent item at a least cost to the publisher but at the requiredsecurity and/or quality level is selected at 510. At 512, the resourcemanager may optionally facilitate pre-fetching of the content item atthe selected resource provider to warm the cache at the resourceprovider. The ability of the selected resource provider to servicerequests for the content item in accordance with the preferencesspecified by the content publisher is monitored at 514, e.g., by aspider process of monitoring module 502 of FIG. 5A. In some embodiments,the availability of the content item at the selected resource provideris also monitored at 514; and if the content item is at some pointdetermined to be unavailable, in some cases process 506 may beredirected to step 512 (not shown in FIG. 5B). A received request forthe content item is redirected, e.g., by director module 504 of FIG. 5A,to the selected resource provider at 516 in the event that the selectedresource provider is able to service the request. In the event that theselected resource provider is unable to service a received request forthe content item, a different resource provider capable of servicing therequest is selected at 510.

FIGS. 6A-6C are block diagrams illustrating manners in which a contentrequest from an end user is redirected by a resource manager to a nodecapable of servicing the request. In the network environments depictedin the examples of FIGS. 6A-6C, the resource consumer comprises acontent publisher that has subscribed to the services of the resourcemanager for servicing user requests for content published by the contentpublisher. When signing up for the services of resource manager 602(e.g., at 404 of process 400 of FIG. 4A), the content publisherspecifies the locations of one or more associated publisher origins 604from which content published by the content publisher can be obtainedand cached at either nodes. A request from user 606 for content (e.g., afile) published by the publisher is directed or redirected to resourcemanager 602. For example, the request may comprise a hyperlink or URLthat redirects to resource manager 602. Resource manager 602 selects anappropriate node 608 to service the request based on the preferencesspecified by the content publisher and redirects the request to node608. In various embodiments, the requested content may be obtained bynode 608 from publisher origin 604 (or from another node at which thecontent is available) either prior to receiving the redirected requestor in response to receiving the redirected request. Node 608 providesthe requested content to user 606, fulfilling servicing of the originalrequest.

In some embodiments, the various redirections of the original requestare transparent to the user. In some embodiments, a set of one or moreinitial requests for a content item may be redirected by resourcemanager 602 to publisher origin 604 and serviced by publisher origin 604(not shown in FIGS. 6A-6C), e.g., when the requested content item hasnet yet been populated at other nodes 608 or when the existence of therequested content item at various other nodes 608 is unknown to resourcemanager 602.

In the example of FIG. 6A, the request is redirected by resource manager602 to a node 608(a) configured (e.g., via process 200 of FIG. 2A) to atleast in part function as a proxy server. Proxy server 608(a) comprisesa confederated node of the network managed by resource manager 602 sinceit has willingly signed up to be a part of the network. In someembodiments, a spider process of the monitoring module of resourcemanager 602 coordinates pre-fetching of a content item at the proxyserver cache before any request for that content item is redirected toproxy server 608(a) so that the request is redirected to a warm cache.Alternatively, proxy server 608(a) may obtain and cache the requestedcontent item in response to receiving a first request for the contentitem or a first request for the content item after a previous copy ofthe content item has been purged from its cache. A cached copy of thecontent item at proxy server 608(a) is deleted from the cache at theexpiration time associated with the cached copy, and a new copy of thecontent item may subsequently be obtained to refresh the cache. Atransaction log for servicing the request is compiled at proxy server608(a) and provided to resource manager 602, e.g., so that it can beused by resource manager 602 to bill the content publisher and reimbursethe provider of proxy server 608(a).

The content publisher may require security between the proxy servercache and the publisher origin. In some embodiments, content is bothtransacted from the publisher origin securely and locally cachedsecurely using an encryption algorithm to prevent spreading of thecontent to nodes configured to serve it and to ensure integrity of thecaches at the nodes. In some embodiments, the software installed on anode to configure it as a proxy server includes a built in shared secretthat is employed to encrypt files that are stored in the local cache, toaccess the remote origin, and to sign transaction logs. Such a securitysystem may also include an auto update mechanism to update the sharedsecret along with monitoring to disable nodes that attempt to tamperwith the log signatures. The transaction logs are signed using theshared secret. Each chunklet of log data sent back to the resourcemanager includes a timestamp and a hash of the entire log chunklet,which includes the shared secret. When the resource manager receives thelog chunklet, it verifies the data by performing the same hash andcompares the received hash with its locally generated hash.

In the example of FIG. 6B, the request is redirected by resource manager602 to a node 608(b) of a caching proxy CDN. In this example, thecontent publisher has a pre-existing contract with the caching proxy CDNand thus requires at least a portion or a specified amount or percentageof its traffic to be serviced by this CDN. The terms of the contentpublisher's contract with the CDN may be specified, for example, withrespect to its account with resource manager 602 (e.g., at 404 ofprocess 400 of FIG. 4A). As previously described, a spider process ofthe resource manager may coordinate ensuring that a content item isavailable at a node before any request for that content item isredirected to that node so that the request is redirected to a warmcache. With respect to the example of FIG. 68, the availability of therequested content item at CDN node 608(b) may be determined via an HTTPHEAD or other appropriate request to CDN node 608(b). If the contentitem does not already exist at the CDN node 608(b), in some cases, thespider may independently request the content item from CDN node 608(b)to warm the cache at the node prior to any actual user requests for thecontent item being redirected to the node. The spider may need toperiodically re-learn the availability of the content item at the CDNnode, e.g., since the content item may be deleted from the cache at theCDN node if it has not been recently served by the CDN node or at itsexpiry time. Alternatively, CDN node 608(b) may obtain and cache therequested content item in response to receiving a first request for thecontent item or a first request for the content item after a previouscopy of the content item has been purged from its cache. CDN node 608(b)comprises a federated node of the network managed by resource manager602 since the caching proxy CDN has been forced to become a part of thenetwork managed by resource manager 602 due to its contract with thecontent publisher. Since CDN node 608(b) does not comprise a resourceprovider node established by resource manager 602, no log data iscompiled and provided to resource manager 602 for servicing the request,and the CDN independently bills the content publisher for servicing therequest. In some embodiments, the caching proxy CDN may choose to jointhe confederation of resource providers managed by the resource manager,e.g., via process 200 of FIG. 2A.

In the example of FIG. 6C, the request is redirected by resource manager602 to a node 608(c) of a storage-based CDN. In this example, thecontent publisher has a pre-existing contract with the storage-based CDNand thus requires at least a portion or a specified amount or percentageof its traffic to be serviced by this CDN. The terms of the contentpublisher's contract with the CDN may be specified, for example, withrespect to its account with resource manager 602 (e.g., at 404 ofprocess 400 of FIG. 4A). The storage-based CDN may only store a subsetof the content published by the content publisher. In some cases, theCDN may only store the most popular content published by the contentpublisher. With respect to a storage based CDN, it is important for theresource manager to verify the availability of a content item at the CDNprior to redirecting any requests for that content item to the CDNbecause if the CDN does not have the requested content item a contentnot found error message (such as an HTTP 404 error message) istransmitted to the user who issued the request, with the resourcemanager remaining unaware that the request was not serviced. Theavailability of a content item at the CDN may, for example, bedetermined by a spider process of the resource manager via an HTTP HEADor other appropriate request. The spider may need to periodicallyre-learn the availability of the content item, e.g., since the contentitem may be deleted from the CDN at its expiry time, which is learned bythe spider from the HEAD request and stored at the resource manager. Insome embodiments, the content publisher is instructed to not remove ordelete a content item from the CDN prior to the expiry time of thecontent item and/or is instructed to notify the resource manager of anysuch action prior to expiry time so that the resource manager can ensurethat a content request is only redirected to the CDN if it has thecontent item without having to interrogate the CDN for availability ofthe content item prior to redirecting every request for the contentitem. With respect to the example of FIG. 6C, the request is redirectedto CDN node 608(c) because the requested content item is known byresource manager 602 to be available at CDN node 608(c). CDN node 608(c)comprises a federated node of the network managed by resource manager602 since the storage-based CDN has been forced to become a part of thenetwork managed by resource manager 602 due to its contract with thecontent publisher. Since CDN node 608(c) does' not comprise a resourceprovider node established by resource manager 602, no log data iscompiled and provided to resource manager 602 for servicing the request,and the CDN independently bills the content publisher for servicing therequest. In some embodiments, the storage-based CDN may choose to jointhe confederation or resource providers managed by the resource manager,e.g., via process 200 of FIG. 2A. By installing the configurationsoftware provided by the resource manager, the nodes of a storage-basedCDN . may be configured to behave as proxy servers, allowing thestorage-based CDN to additionally operate as a caching proxy CDN.

As described in the examples of FIGS. 6B-6C, the resource manager mayredirect traffic to a CDN based on the preferences and usageinstructions specified by the content publisher. In some embodiments,the resource manager manages publisher contracts with a plurality ofdifferent CDNs and redirects publisher traffic based on the terms of thecontracts, e.g., in a manner that minimizes content delivery costs tothe publisher. For example, the resource manager may minimize or atleast reduce a publisher's content delivery costs by intelligently usingthe bandwidth of one or more contracted CDNs that bill with the 95^(th)percentile billing model. With CDNs that use such burstable billingmodels, the resource manager may also maximize use of free burstablebandwidth, which on a monthly billing cycle translates to up to 36 hoursof free bandwidth per CDN.

In various embodiments, any appropriate billing and settlement model maybe employed in the system of resource consumers and resource providersmanaged by the resource manager. The resource manager keeps track of theparticipants involved in each transaction as well as details of thetransaction, e.g., via the log data received at the conclusion of eachtransaction from the resource provider. During a (e.g., monthly)settlement process, payments are received from resource consumers anddistributed to resource providers as applicable. The resource managermay take a small transaction fee or a small percentage of the paymentfor facilitating the transaction. In some cases, the resource managermay track and bill for the total number of managed requests. Inaddition, the resource manager may bill for special services, such ascache warming, use of certain protocols, etc. In some embodiments, an ala carte billing model may be employed where each type of resourcemanaged by the resource manager is billed on a per transaction basis, afeature basis, or a statistics basis. Alternatively, various types ofresources may be bundled together to create packages and differentservice level offerings. With respect to content delivery, a resourceconsumer may be billed based on the volume or total bytes of trafficserved. In such cases, for example, the cost of each transaction may becomputed from the product of the price per byte at delivery time and thetotal bytes delivered for the transaction, which values may be obtainedfrom the log data of the transaction provided by the resource providerat the time of the transaction. In some such cases, the resource managermay add a small surcharge to the price per byte or may bill a flat feefor facilitating the transaction. In other embodiments, the 95thpercentile value of a resource consumer may be determined across allresource providers over a billing cycle, e.g., by aggregating the datafrom the transaction logs provided by the resource providers. In somesuch cases, the 95th percentile value is multiplied by the fraction ofthe total traffic over the billing cycle that a particular resourceprovider delivered to obtain the bandwidth value billable by thatresource provider. In such cases, the resource manager may take a smallpercentage of the amount billed by the resource provider.

Although crowd based content delivery is described in many of theexamples provided herein, the resource manager may be similarly employedto facilitate any crowd based computing platform. For example, in someembodiments, the resource manager may facilitate crowd based storage bywhich content items are replicated for storage across the crowd. In someembodiments, the resource manager may facilitate crowd based computingby which compute modules are distributed across the crowd to performtasks such as video compression and encoding, encryption cracking,distributed web hosting and/or application execution, etc. In someembodiments, the resource manager may facilitate military purposes suchas distributed network defense and offense mechanisms. For example, as adefense mechanism, the crowd may be employed as a distributed DDoSfilter to protect from a DDoS attack. Likewise, as an offense mechanism,the crowd may be employed to generate such attacks.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

1-22. (canceled)
 23. A method comprising: receiving, at a resourcemanager, a preference of a content publisher; receiving, at the resourcemanager, from the content publisher an indication that the resourcemanager is to redirect a request for a content item; monitoring, at theresource manager, a state of a network; selecting a resource provider inthe network capable of servicing the request based at least in part onthe state of the network and the preference of the content publisher;and redirecting the request for the content item to the resourceprovider.
 24. The method of claim 23, wherein monitoring a state of anetwork is based on performance statistics associated with quality ofservice and end user experience.
 25. The method of claim 24, wherein theperformance statistics comprise one or more of an average throughput, amaximum throughput, a DNS lookup time, a time to first connection and adownload time.
 26. The method of claim 24, wherein performancestatistics is reported to the resource manager by a dedicated monitoringserver that is placed across the network.
 27. The method of claim 24,wherein performance statistics is reported to the resource manager by aplug-in that is included in a source.
 28. The method of claim 23,wherein monitoring a state of a network is based on log data compiled bythe resource provider at the conclusion of each transaction.
 29. Themethod of claim 23, wherein monitoring a state of a network comprisesfurther monitoring availability of the resource provider to service therequest.
 30. The method of claim 29, wherein monitoring a state of anetwork comprises further monitoring a location of the content item inthe network.
 31. The method of claim 30, wherein the location of thecontent item is determined by interrogating the network using aninterrogation methodology.
 32. The method of claim 23, furthercomprising coordinating, at the resource manager, a pre-fetching of thecontent item at the resource provider to warm the cache before therequest is redirected.
 33. The method of claim 23, wherein the selectinga resource provider is further based at least in part on past trafficredirected to the resource providers to prevent the resource providerfrom becoming overloaded and to load balance a plurality of availableresource providers.
 34. A computer system for handling requestcomprising: a non-transitory storage device having embodied therein oneor more routines; and one or more processors coupled to thenon-transitory storage device and operable to execute the one or moreroutines to perform a method comprising: receiving a preference of thecontent publisher; receiving from a content publisher an indication thatthe resource manager is to redirect a request for a content item;monitoring a state of a network; selecting a resource provider capableof servicing the request based at least in part on the health of thenetwork and the preference of the content publisher; and redirecting therequest for the content item to the resource provider.
 35. The computersystem of claim 34, wherein monitoring health of a network is based onperformance statistics associated with quality of service and end userexperience.
 36. The computer system of claim 35, wherein the performancestatistics comprise one or more of an average throughput, a maximumthroughput, a DNS lookup time, a time to first connection and a downloadtime.
 37. The computer system of claim 35, wherein performancestatistics is reported by a dedicated monitoring server that is placedacross the network.
 38. The computer system of claim 35, whereinperformance statistics is reported by a plug-in that is included in asource.
 39. The computer system of claim 34, wherein monitoring healthof a network is based on log data compiled by the resource provider atthe conclusion of each transaction.
 40. The computer system of claim 34,wherein monitoring a state of a network comprises further monitoringavailability of the resource provider to service the request.
 41. Thecomputer system of claim 40, wherein monitoring a state of a networkcomprises further monitoring a location of the content item in thenetwork.
 42. The computer system of claim 41, wherein the location ofthe content item is determined by interrogating the network using aninterrogation methodology.
 43. The computer system of claim 34, furthercomprising coordinating a pre-fetching of the content item at theresource provider to warm the cache before the request is redirected.44. The computer system of claim 34, wherein the selecting a resourceprovider is further based at least in part on past traffic redirected tothe resource providers to prevent the resource provider from becomingoverloaded and to load balance a plurality of available resourceproviders.